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ABSTRACT 

This paper discusses the benefits of conducting multi-system integration testing of space 
flight elements in lieu of merely shipping and shooting to the launch site and launching. 
“Ship and shoot” is a philosophy that proposes to transport flight elements directly from 
the factory to the launch site and begin the mission without further testing. Integration 
testing, relevant to validation testing in this context, is a risk mitigation effort that builds 
upon the individual element and system levels of qualification and acceptance tests, 
greatly improving the confidence of operations in space. The International Space Station 
Program (ISSP) experience is the focus of most discussions from a historical perspective, 
while proposed integration testing of the Constellation Program is also discussed. The 
latter will include Multi-Element Integration Testing (MEIT) and Flight Element 
Integration Testing (FEIT). 
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HISTORY 

Ship and shoot has always been a goal as seen by program managers from the Redstone 
Rocket Program in the mid 1950’s through Mercury, Gemini, and the Apollo Programs, 
including the Saturn family, IB’s and V’s, to the current Space Shuttle Program. A 
companion goal of ship and shoot has been to utilize a set of factory checkout equipment 
at the launch site. There also have been program proponents to use the same test 
procedures at the launch site as those used at the factory. Each of these goals may seem 
feasible at the beginning and during the early phases of the program. However, the real 
world of delivering flight hardware and Ground Support Equipment (GSE) to the launch 
site has provided sufficient negative experience to hamper the achievement of these 
goals. 



Another point that emphasizes the need for integration verification is revealed on the 
Shuttle Program. As of 2001 , this Program no longer required the contractor to conduct 
integrated hazard analysis. They were only required to perform analysis at component 
and element levels, and not required to evaluate the Shuttle as a whole. Exclusively 
using a “bottoms-up” verification approach at the component level means that a “top- 
down” verification cannot be effectively supported, which is needed to determine risk 
trends and identify potentially harmful interactions between systems. 

A prevalent experience has been that the flight hardware has been shipped to the launch 
site prior to being fully completed, tested and accepted. History says that this condition 
has been promoted by both the program and the contractor to meet program advertised 
ship dates as well as the contractor who has the ship date as a metric milestone in the 
contract. There is a host of factors that can drive failure of the mission including 
hardware arriving at the launch site late with “open work”, hardware that has never seen 
its ground or flight interfaces, factory checkout only focusing on getting by to meet 
contractual milestones, launch site checkout focusing on hardware working with all 
interfaces, launch site checkout being the last chance to verify quality of hardware for 
flight, and launch site inspections discovering many discrepancies such as bent connector 
pins, loose fluid fittings, tools left in compartments, damaged wire bundles, and debris. 
These factors cause inherent safety risks as well as technical difficulties. 

The ISSP baseline was originally intended to be ship and shoot. Factory level testing and 
element interface verifications at the subsystem-level, and interface analysis were all that 
was planned. In the spring of 1996, the testing evolution added the ISS Flight 2A 
Interface Verification Test (IVT), and various Program changes expanded requirements 
for Shuttle Avionics Integration Laboratory (SAIL), Cargo Element, and IVT. 

Availability of elements at the launch site created a feasible opportunity to test multiple 
elements together, and it was decided that ISS elements would not be launched until 
extensive integration testing was performed prior to launch of these elements. The notion 
of verification on the ground was presented and accepted by the Program due to the 
criticality of the on-orbit functionality, including safety concerns for the crew. Multi 
Element Integration Test (MEIT)/Integrated Systems Test (1ST) verified the operation of 
the flight element in an environment that was as flight-like as possible. Space Station 
element-to-element interface capability was verified as well as systems end-to-end 
operability with hardware and software. MEIT also verified limited on-orbit scenario 
system compatibility. The Mission Sequence Test (MST), also known as “end-to-end” 
integration testing, included a full-up configuration with the Mission Control Center, 
Tracking and Data Relay Satellite System (TDRSS) satellite communication, and the 
“station” operating on the ground. This allowed for a complete flight-like configuration 
that utilized critical communication links. 


ISSP MEIT TEST CONFIGURATIONS 



Three Multi-Element Integration Tests and one Integration Systems Test (1ST) were 
conducted for the ISS. Within each MEIT, there were several ISS test configurations. At 
a high level MEIT1 consisted of US Lab, Z1 truss, P6 truss, and a Nodel emulator. 
MEIT2 consisted of SO truss/Mobile Transporter/Mobile Base System, SI truss, PI truss, 
P3 truss, P4 truss, and a US Lab emulator. MEIT2 boasted the largest payload test in 
history, with over 3000 cables. MEIT3 included the Japanese Experiment Module, 
Node2, and the US Lab emulator, which morphed into the ISS Flight Emulator as it was 
augmented to represent multiple ISS elements throughout the program. Finally, the 
Node2 1ST was comprised of Node2 and US Lab and Nodel emulators, as part of the ISS 
Flight Emulator. 


ISSP MEIT MAJOR FINDS 

During the multi-element integration tests conducted for ISS, significant problems that 
could only have been found in an integrated configuration with the.multi-element flight 
hardware and software were identified and corrected. These problems would have 
seriously impacted Station assembly operations. Many other problems have also been 
found and corrected or had program workarounds prior to flight. Over 5000 verification 
requirements were accomplished. 

Examples of major finds during ISS MEIT/IST testing are as follows. 

• The Assembly Power Converter Unit (APCU) was redesigned after power losses 
due to breaker trips during the ISS P6 element activation on the ground before 
flight. The impact would have prevented the activation of P6 on-orbit, driving a 
shuttle re-flight of P6. 

• During MEIT2, video lines were found to be swapped between United States On- 
Orbit Segment (USOS) Trailing Umbilical System (TUS) and the Canadian 
Mobile Base System (MBS). This would have operationally impacted manual 
routing of video signals from the Space Station Remote Manipulator System 
(SSRMS) to Robotics Work Station (RWS), thus requiring a time-consuming 
Extra Vehicular Activity (EVA) to swap these two harnesses. 

• Multiplexer/De-Multiplexer (MDM) computer task overruns resulted in US Lab 
Command and Control (C&C) computer switchovers. This would have resulted 
in a loss of vehicle commanding and vehicle health visibility to the crew and 
ground. 

• The initial US Lab element critical activation took 36 hours, and the requirement 
was less than two hours. The impact could have resulted in the loss of the US Lab 
element during on-orbit activation due to thermal loading timeline requirements. 

• Mass property units were mismatched between the RWS and the US Lab C&C, 
which could have resulted in attitude errors, adding propellant usage. 

• The Video Processing Unit design was changed after jittery orbiter video signals 
were found in the US Lab. 

• External (EXT) MDM computer task overruns were discovered upon enabling 
planned on-orbit applications, impacting the start of the primary cooling system. 



• A SSRMS/RWS software problem that would have prevented the first SSRMS 
walk-over attempt was corrected prior to flight. 

• An electrical short was corrected on the Flight Support Equipment Grapple 
Fixture (FSEGF), preventing the SSRMS power-up; a minimal impact would 
have been an unscheduled EVA to identify and correct the problem, while a 
maximum impact would have been a SSRMS shuttle re-flight. 

• It was discovered that redundant video synchronizations supplied from the US 
External Video Switch Unit (EVSU) were not carried through the Canadian 
Mobile Base System (MBS), causing a significant impact due to total loss of 
video from SSRMS/MBS cameras to the RWS. 

• Control Moment Gyros (CMGs) powered down unexpectedly during a US Lab 
C&C computer switchover; the on-orbit impact would have been regaining 
control of the US On-orbit Segment (USOS) attitude and using more propellant. 

• Other finds worth mentioning are a US Lab C&C Small Computer System 
Interface (SCSI) bus cabling redesign due to 1553 errors; audio downlink quality 
issues for accented or foreign language; Solar Array Rotating Joint 
(SARJ)/Trailing Radiator Rotating Joint (TRRJ) auto-tracking failure due to a 
GNC software problem; and a US Lab C&C computer failure during sync to GPS 
time. 

• Hundreds of software glitches and problems were found and corrected. 
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Figure 1. ISSP MEIT Configurations 





OTHER MEIT BENEFITS 


An invaluable factor is the experience that flight controllers, station crew, International 
Partners, and engineers have gained from participation. These integration tests have also 
paced the development of flight activation procedures. A significant number of 
deviations have been incorporated into flight procedures, preventing failed on-orbit 
activation and checkout activities. The cumulative effect of identifying and correcting all 
the integrated hardware/software/procedure problems before flight has saved the ISS 
Program from major on-orbit failures. Also, in addition to operational impacts, cost and 
schedule have been saved. Most of these problems would have been far more expensive 
and riskier to correct on orbit. 


FURTHER OBSERVATIONS 

There is an interesting point in the history of ISSP MEIT testing. Metrics show that later 
testing such as Node 2 1ST and MEIT3 revealed significant improvements in the software 
products as well as the hardware delivered from the factory. For MEIT1, a total of 884 
Interim Problem Reports (IPRs) were taken, while MEIT2 had 334, and MEIT3 had 64. 
Some anomalies should have been discovered at the factory level, during qualification 
and acceptance testing. Not all problems were pure integration testing finds. One could 
conclude that a more rigorous test program at the element/system level is important for 
integration testing, to avoid large-scale configurations to find standalone element 
problems. As a result of using mature hardware and software, integration testing 
demonstrates on-orbit success for critical flight operations on the ground versus on-orbit, 
in a cost and schedule effective manner. Therefore, it is notable that the incorporation of 
requirements, the hardware manufacturing and qualification stages, as well as software 
development stages are equally as critical as integration and test of the entire system for a 
mission. 

Simulations and emulations of unavailable elements necessary to meet the requirements 
of the configuration are also important. MEIT developed an ISS Flight Emulator which 
emulated vital elements for testing purposes. The Software Development Integration 
Laboratory (SDIL) in Houston, Texas, representing the entire station, was also invaluable 
for providing hardware and software integration. These simulations allowed verification 
of the software, the opportunity to perform critical procedure validation prior to testing 
with flight hardware, reducing the risk of damage to hardware and personnel, and the 
ability to re-create and troubleshoot on-orbit anomalies. Simulations and emulations are 
a must for successful testing at various levels of development. Finally, a further 
improvement to testing should include autonomous information systems, providing real- 
time payload/vehicle configuration, automatic tracking of test requirements, and 
instantaneous quality audits prior to the mission. 

If integration testing is planned early enough, would it change the development of the 
hardware design? Possibly hard mating of the elements could have been accomplished 



with ISS elements, rather than building and procuring expensive jumper cables. The 
hardware’s structural limitations did not allow for orientations necessary to easily 
perform hardware mates for full functional testing. It should be evaluated during the 
requirements and design stages if modifications could be accomplished to drastically 
improve configurations to enable ease of testing and reduce costs. 

Integration testing and verification requirements also have significant impacts on facility 
and GSE development. A single set of standards and specifications should be provided 
for everyone to utilize during fabrication and assembly. The GSE and facility should be 
verified thoroughly and certified prior to the arrival of flight hardware. There have been 
too many instances where the GSE has failed and consumed valuable test time to resolve 
before the flight article was tested. Having a solid, defendable plan for verification - 
whether it’s factory-level checkout or full functional integration testing - should be 
defined early enough to have a positive affect on the design of ground and flight 
systems. The design of GSE, including access, handling, processing and checkout 
systems, would be enhanced if the test requirements are identified and incorporated early 
in the development process. 

If the flight hardware has been accepted and shipped to the launch site as elements or 
systems, and fully integration tested, it will still require validation interface testing with 
interconnect cabling with the launch vehicle. Although the use of simulators (emulators) 
to replace unavailable hardware assists in earlier testing, the actual interfacing of the 
mating hardware must be tested prior to launch to assure that all systems are functional 
and have the required margins for flight. 

The on-board software should be baselined prior to the start of integration testing. 
Software development has typically lagged behind the manufacturing of the hardware. 

The ISSP has shown repeatedly that a final version software load is one of the last 
processes before the element is taken to the launch pad. And then a final power up of all 
components must be conducted to prove the new software load. Early MEIT testing was 
not afforded the opportunity to run with Flight Qualification Tested (FQT) software. 
Integration testing should not be the primary test of the flight software. It should be fully 
integration tested and qualified in a software laboratory prior to an integration test with 
the hardware. 

“For systems requiring incremental assembly where elements involve distributed end-to- 
end subsystems in low earth orbit or beyond earth orbit, it is prudent to conduct multiple- 
element integration testing prior to launch. Use of approaches such as testing elements in 
logical groupings with appropriate fidelity emulations of interfaces is acceptable. The use 
of emulators is, however, less desirable than testing the actual hardware, and additional 
care needs to be exercised in the development of interface controls to ensure the 
emulators reflect the true "as built" configuration of the system. Testing is carried out 
with software possessing flight functionality and flight hardware in flight configuration. 
Priority is given to interface validations of hardware and hardware/software interaction. If 
applicable, end-to-end testing of command and telemetry links between the control 



center(s) and the vehicle can be accomplished.” (NPG 8705.2 Human-Rating 
Requirements and Guidelines for Space Flight Systems 2005) 

A final thought is the notion to use risk analysis to determine if integration testing should 
be performed. However, the very nature of the inability to predict everything that could 
potentially occur with multiple elements, modules, and systems, yields this concept as 
flawed. When systems cross organizational, contractual, or geographic boundaries, 
integration testing becomes imperative. “The risks inherent in these technical systems 
are heightened when they are produced and operated by complex organizations that can 
also break down in unanticipated ways”. (CAIB Report 2003). 


TRANSITIONING THE LEGACY TO THE CONSTELLATION PROGRAM 

Integration testing is defined as testing that occurs between two or more individual 
Constellation systems to verify the interfaces between those systems, and to validate the 
integrated systems’ functionality and interoperability. Integration testing applies to the 
in-space flight and integrated vehicle stack configurations including multi-element 
integration testing (MEIT), flight element integration testing (FEIT), and interface 
verification testing (IVT). 

For clarification, it should be noted that for the Constellation Program, systems are 
comprised of elements, such as the Service Module, the Crew Module, the Launch Abort 
System are all elements of the Crew Exploration Vehicle (CEV) (Orion spacecraft). The 
First Stage and Upper Stage are elements of the Crew Launch Vehicle (CLV) (Ares I 
rocket). Examples of the system would be the CEV or the CLV, while for ISSP, the US 
Lab, Node2, and the P6 truss are elements. Figure 2 depicts the system, element, and 
subsystem breakdown of the Constellation architecture. The terms “MEIT” and “FEIT” 
are legacy terminology and have been adopted by the Constellation Program. For 
Constellation, the terms are used to indicate multi-“system” and flight “systems” tests 
instead of multi-“element” and flight “element” tests. 
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Figure 2. Constellation Program Architecture Systems, Elements, and Subsystems. 


MEIT 

As already understood by the context of the paper, MEIT is an integration test between 
two or more flight systems that will be launched on separate vehicles and integrated 
together for the first time in space. Interfaces between the flight systems, mission 
systems, and other appropriate Constellation Program (CxP)/ISSP and external systems 
may also be tested as part of each MEIT. 

Primary objectives of MEIT for this program are to demonstrate interoperability, 
functionality, and stability of the flight systems, elements, and subsystems as an 
integrated “in-space” vehicle assembly on the ground before they are assembled in space 
for the first time. Critical mission sequence activities and flight procedures will be 
validated prior to their first-time execution in flight. 

A secondary objective of MEIT is to collect functional and performance data from the 
integrated flight systems to support validation of the different interface test tool sets. 







emulators, and simulations used by the projects in accepting future serial numbers of 
those flight systems. 


MEIT also serves as a training ground for flight crew and trainers. The Mission 
Operations Division (MOD)/Mission Control Center (MCC) personnel gain great 
experience during these activities. 

A critical point is that MEITs are executed after all of the system-level requirements have 
been satisfied by the respective project offices, and they have declared their system 
designs as verified for flight. All System Requirement Document (SRD) and Interface 
Requirement Document (IRD) verification requirements have been closed, thus rendering 
MEIT a validation test, operating as closely as possible to flight scenarios. CxP MEITs, 
performed prior to first crewed missions of the ISS and Lunar blocks, are one-time tests 
with a potential for follow-on regression testing. 

At the writing of this paper, two planned MEITs are documented as depicted in Figures 3 
and 4. 
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Figure 3. Crew Exploration Vehicle (CEV) to International Space Station (using ISS 
emulation) - Orion/ISS, and includes Mission Control Center (MCC), TDRSS, ground 
station. 



ME IT 2 Lunar Block 



Figure 4. CEV to Lunar Lander/Earth Departure Stage (EDS) (using EDS emulation) - 
Orion/Lunar Surface Access Module (LSAM), and includes Mission Control Center 
(MCC), TDRSS, ground station. 

The Lunar block MEIT may contain multiple configurations since it is intended to 
validate inter-system interfaces and functionality in both the launch and Low Earth Orbit 
(LEO) configuration (CEV to LSAM/EDS), as well as the post Trans-Lunar Injection 
(TLI) configurations after EDS jettison (CEV to LSAM), and then CEV to LSAM ascent 
stage, to validate the configuration after the ascent stage returns from the moon and re- 
mates with the CEV without the descent stage. 


FURTHER DEVELOPMENT OF MEIT1 (ISS-CEV) TEST OBJECTIVES 

As the program delves further into test objectives development, more detailed 
requirements will emerge. Objective examples are as follows: 

• The program will verify audio quality, frequency response, and harmonic 
distortion for all audio signals in mated configuration (air-air, air-ground, 
intercom, page key, page voice) 

• Acceptable video quality and signal characteristics for all baseband links between 
CEV and ISS 

• Audio quality, frequency response, and harmonic distortion for both directions 
over open loop CEV-ISS S-Band RF links 

• Packet loss testing of open loop CEV-ISS S-Band RF link with signal level set to 
minimum and maximum limits for both forward and return links 


• Peak power bus currents for each feed in mated configuration to not exceed 
maximum value for both active and quiescent modes 

• Transient, steady state, ripple, noise, and source/load impedance comply with 
power quality standards 

• Simulated on-orbit rendezvous and docking operations (signal acquisition, 
Guidance, Navigation and Control (GNC) moding, command path switching) 

• Simulated on-orbit CEV state/mode transitions (active to quiescent, quiescent to 
active) 

• Simulated on-orbit undocking and separation operations 

• Health status software upload to CEV 

• First flight to station (Orion 1) demonstration of proximity operations capability 

• Capability to support dual CEV docked and undocked operations 

• CEV reception and process of commands generated from Portable Computer 
System (PCS) laptops and commands generated from MCC-H and transmitted via 
ISS S-Band 

• Proper file/memory transfer between CEV and local ISS C&DH systems (PCS, 
C&C mass storage device (MSD), Space Station Complex (SSC)) 

• Proper file/memory transfer between CEV and MCC-H via ISS S-Band 

• CEV telemetry processes and displays on PCS laptops 

• CEV telemetry received via ISS S-Band can be processed and displayed at MCC- 
H 

• CEV acquisition and synchronization to time broadcast from ISS 

• IP packet and 1553 message loss for all Gigabit Ethemet/1553 channel 
combinations in mated configuration 

• IP packet transmission routing and packet loss between CEV and ISS Integrated 
Station LAN 

More objectives will be developed, and from the objectives, many more detailed test and 
verification requirements will result. 


FEIT 

Flight Element Integration Test is defined as testing between new or significantly 
modified systems and elements being assembled into an integrated launch vehicle for the 
first time. Interfaces between the flight systems, ground systems, mission systems, and 
other appropriate CxP and external systems may also be tested as part of each FEIT. 

The primary objectives of FEIT are to demonstrate the interoperability, functionality, and 
stability of the flight systems, elements, and subsystems as an integrated “launch vehicle” 
stack assembly prior to the first operational test flight of that particular launch vehicle 
configuration. Critical mission sequence activities and flight procedures will be validated 
prior to their first-time execution. 



A secondary objective is to collect functional and performance data from the integrated 
flight systems to support validation of the different interface tool sets, emulators, and 
simulations used by the projects in accepting future serial numbers of those flight 
systems. 

As with MEIT, FEITs are executed after all of the system-level requirements have been 
satisfied by the respective project offices and those projects declare their systems designs 
as verified for flight. FEITs are planned as part of the assembly and processing of the 
integrated launch vehicle. FEITs are planned for every test flight, with the exception of 
the Aresl-X test flight, which is a demonstration flight to inform the design of the 
vehicle. 


FEITs are planned as follows: 

• For Aresl (ISS Block): Crew Exploration Vehicle/Crew Launch Vehicle 
(CEV/CLV) - Orion/Aresl, and includes Mission Control Center (MCC), Launch 
Control Center (LCC), TDRSS, ground station, ground systems. 

• For Ares V (Lunar Block): Lunar Lander/Earth Departure Stage/Cargo Launch 
Vehicle (LSAM/EDS/CaLV) - LSAM/EDS/Ares V, and includes Mission 
Control Center (MCC), Launch Control Center (LCC), TDRSS, ground station, 
ground systems. 


INTERFACE VERIFICATION TESTING (IVT) 

Interface Verification Testing (IVT) performs mechanical, data, and electrical interface 
verification and validation between two or more flight systems after these systems have 
been mated. It consists of the minimal set of activities to confirm the integrity of all 
physical and functional interfaces between systems involved for each specific vehicle 
stack configuration. IVTs are performed as a subset of the FEITs during the early flight 
tests and then performed for every flight after Full Operational Capability (FOC) has 
been obtained for both flight and ground systems. This is the testing that will occur for 
all nominal missions for the duration of the program. 
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Figure 5. Validation of Interfaces Between Systems and CxP and External Systems. 



A DIFFERENT TESTING APPROACH FOR CONSTELLATION 


While testing is usually recognized as an important part of a project, it does not typically 
receive the attention that is needed until late in the development flow, nor is it afforded 
the required funds early on and throughout the life of the program. “Testing in general 
does not receive as much attention as it should in order to ensure a project’s success. First, 
testing typically does not receive enough emphasis during the initial planning phases. Life- 
cycle cost estimating is another area that should be enhanced and used in making test 
decisions. When the cost of on-orbit repairs are considered in the decision, it often becomes 
apparent that performing a test is money well spent in the long run.” (Britton and Schaible 
2003, 63, 65). The Constellation Program has shifted the paradigm by developing a 
separate test and verification organization, and developing the strategies, concepts, and 
guidelines at the beginning of the program. This includes development of a correlating 
verification requirement for every requirement at each design level. Furthermore, a 
building block approach, from early development to qualification, acceptance, validation 
integration testing, and flight testing is being implemented. Therefore, the schedule, 
budget, and technical requirements for integration testing are being understood and 
accepted early in the program. This culture change will drive mission success. 


CONCLUSION 

In conclusion, the examples stated in this paper, describing the type of anomalies 
encountered at the launch site, are well-documented evidence of problems that could 
have only been found in an integrated configuration, as well as other problems that 
should have been found in stand-alone testing at the factory. Emphasis should remain 
upon a fully completed and verified element or spacecraft leaving the factory for delivery 
to the launch site, while subsequently performing an end-to-end multi-element integration 
test and flight element integration test of the entire system after delivery to the launch site 
prior to its’ journey into space. To ensure a successful mission, it is imperative that time 
in the schedule and budget allowances permit this testing. The integration testing process 
has evolved into an efficient, cost effective means of ensuring risk mitigation to the ISSP, 
and will accomplish the same goals for the Constellation Program. 

"Standing alone, components may function adequately, and failure modes may be 
anticipated. Yet when components are integrated into a total system and work in concert, 
unanticipated interactions can occur that can lead to catastrophic outcomes". (CAIB 
Report 2003) 
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